Field of Invention
Embodiments of the present invention generally relate to A/B testing, and more specifically to techniques for improving the results of an A/B test through the use of stratified sampling.
Description of Related Art
Web pages on modern websites, such as websites available on the Internet, are powerful communication tools for disseminating information regarding current events, organizations, and other items of interest to website visitors. Web pages can also provide targeted advertising of products and services. The content of web pages used to convey this information and deliver these advertisements varies considerably. Selections regarding style, graphics, and layout impact visitors' responses to the information offered on the web page, and seemingly small changes in web page appearance can sometimes dramatically change the effect of the web page on visitors. Certain websites generate revenue by way of a subscription model, in which customers periodically pay a fee to renew their subscription to the website. In such a website, selections regarding the style, graphics, and layout of the website, as well as selections of algorithms used by the website, can affect visitors' decisions with respect to their subscriptions or potential subscriptions.
The effectiveness of differing combinations of web page content can be gauged by measuring visitor responses to varying web page content. Through experimentation, effective website content can be identified. One technique for evaluating various website content is through the use of multivariate testing (also referred to as “split testing” and “A/B testing”). Generally, in online multivariate testing, different variations of a user experience on a website are tested by presenting each user experience to a respective group of users when the users access the website. For instance, a user experience could be a layout of content on the online website, where each variation represents a distinct arrangement of the content. As another example, a user experience could be a content selection algorithm that is used to select content that is included on a webpage of the website. As yet another example, the user experience could include an online advertisement, a graphic or more generally any component of a webpage that has one or more attributes (e.g., color, size, shape, location, text, etc.,) that can be varied. The different variations could then be presented to users, and the users' actions could be tracked (e.g., using user subscription information such as a login ID, using user IDs generated from the users' browser attributes and possibly other user device attributes, etc.). The captured actions may be used to determine the effectiveness of different variations of the user experience.